{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <center/>同步训练和验证模型体验"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 概述"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在面对复杂网络时，往往需要进行几十甚至几百次的epoch训练。而在训练之前，往往很难掌握在训练到第几个epoch时，模型的精度能达到满足要求的程度。所以经常会采用一边训练的同时，在相隔固定epoch的位置对模型进行精度验证，并保存相应的模型，等训练完毕后，通过查看对应模型精度的变化就能迅速地挑选出相对最优的模型，本文将采用这种方法，以LeNet网络为样本，进行示例。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "整体流程如下：\n",
    "1. 数据集准备。\n",
    "2. 构建神经网络。\n",
    "3. 定义回调函数EvalCallBack。\n",
    "4. 定义训练网络并执行。\n",
    "5. 定义绘图函数并对不同epoch下的模型精度绘制出折线图。\n",
    "\n",
    "> 本文档适用于CPU、GPU和Ascend环境。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 数据准备"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据集的下载"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下载并解压数据集数据。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-12-01 14:57:18--  https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/datasets/MNIST_Data.zip\n",
      "Resolving proxy-notebook.modelarts-dev-proxy.com (proxy-notebook.modelarts-dev-proxy.com)... 192.168.0.172\n",
      "Connecting to proxy-notebook.modelarts-dev-proxy.com (proxy-notebook.modelarts-dev-proxy.com)|192.168.0.172|:8083... connected.\n",
      "Proxy request sent, awaiting response... 200 OK\n",
      "Length: 10754903 (10M) [application/zip]\n",
      "Saving to: ‘MNIST_Data.zip’\n",
      "\n",
      "MNIST_Data.zip      100%[===================>]  10.26M  48.1MB/s    in 0.2s    \n",
      "\n",
      "2020-12-01 14:57:19 (48.1 MB/s) - ‘MNIST_Data.zip’ saved [10754903/10754903]\n",
      "\n",
      "Archive:  MNIST_Data.zip\n",
      "   creating: MNIST_Data/test/\n",
      "  inflating: MNIST_Data/test/t10k-images-idx3-ubyte  \n",
      "  inflating: MNIST_Data/test/t10k-labels-idx1-ubyte  \n",
      "   creating: MNIST_Data/train/\n",
      "  inflating: MNIST_Data/train/train-images-idx3-ubyte  \n",
      "  inflating: MNIST_Data/train/train-labels-idx1-ubyte  \n",
      "./datasets/MNIST_Data/\n",
      "├── test\n",
      "│   ├── t10k-images-idx3-ubyte\n",
      "│   └── t10k-labels-idx1-ubyte\n",
      "└── train\n",
      "    ├── train-images-idx3-ubyte\n",
      "    └── train-labels-idx1-ubyte\n",
      "\n",
      "2 directories, 4 files\n"
     ]
    }
   ],
   "source": [
    "!wget -N https://obs.dualstack.cn-north-4.myhuaweicloud.com/mindspore-website/notebook/datasets/MNIST_Data.zip\n",
    "!unzip -o MNIST_Data.zip -d ./datasets\n",
    "!tree ./datasets/MNIST_Data/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 数据集的增强操作"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "下载下来后的数据集，需要通过`mindspore.dataset`处理成适用于MindSpore框架的数据，再使用一系列框架中提供的工具进行数据增强操作来适应LeNet网络的数据处理需求。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import mindspore.dataset as ds\n",
    "import mindspore.dataset.vision.c_transforms as CV\n",
    "import mindspore.dataset.transforms.c_transforms as C\n",
    "from mindspore.dataset.vision import Inter\n",
    "from mindspore import dtype as mstype\n",
    "\n",
    "def create_dataset(data_path, batch_size=32, repeat_size=1,\n",
    "                   num_parallel_workers=1):\n",
    "    # define dataset\n",
    "    mnist_ds = ds.MnistDataset(data_path)\n",
    "\n",
    "    # define map operations\n",
    "    resize_op = CV.Resize((32, 32), interpolation=Inter.LINEAR)  \n",
    "    rescale_nml_op = CV.Rescale(1 / 0.3081, -1 * 0.1307 / 0.3081) \n",
    "    rescale_op = CV.Rescale(1/255.0, 0.0) \n",
    "    hwc2chw_op = CV.HWC2CHW() \n",
    "    type_cast_op = C.TypeCast(mstype.int32) \n",
    "\n",
    "    # apply map operations on images\n",
    "    mnist_ds = mnist_ds.map(operations=type_cast_op, input_columns=\"label\", num_parallel_workers=num_parallel_workers)\n",
    "    mnist_ds = mnist_ds.map(operations=[resize_op,rescale_op,rescale_nml_op,hwc2chw_op],\n",
    "                            input_columns=\"image\", num_parallel_workers=num_parallel_workers)\n",
    "\n",
    "    # apply DatasetOps\n",
    "    buffer_size = 10000\n",
    "    mnist_ds = mnist_ds.shuffle(buffer_size=buffer_size)\n",
    "    mnist_ds = mnist_ds.batch(batch_size, drop_remainder=True)\n",
    "    mnist_ds = mnist_ds.repeat(repeat_size)\n",
    "    \n",
    "    return mnist_ds"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 构建神经网络"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "LeNet网络属于7层神经网络，其中涉及卷积层，全连接层，函数激活等算法，在MindSpore中都已经建成相关算子只需导入使用，如下先将卷积函数，全连接函数，权重等进行初始化，然后在LeNet5中定义神经网络并使用`construct`构建网络。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "import mindspore.nn as nn\n",
    "from mindspore.common.initializer import Normal\n",
    "\n",
    "\n",
    "class LeNet5(nn.Cell):\n",
    "    \"\"\"Lenet network structure.\"\"\"\n",
    "    # define the operator required\n",
    "    def __init__(self, num_class=10, num_channel=1):\n",
    "        super(LeNet5, self).__init__()\n",
    "        self.conv1 = nn.Conv2d(num_channel, 6, 5, pad_mode='valid')\n",
    "        self.conv2 = nn.Conv2d(6, 16, 5, pad_mode='valid')\n",
    "        self.fc1 = nn.Dense(16 * 5 * 5, 120, weight_init=Normal(0.02))\n",
    "        self.fc2 = nn.Dense(120, 84, weight_init=Normal(0.02))\n",
    "        self.fc3 = nn.Dense(84, num_class, weight_init=Normal(0.02))\n",
    "        self.relu = nn.ReLU()\n",
    "        self.max_pool2d = nn.MaxPool2d(kernel_size=2, stride=2)\n",
    "        self.flatten = nn.Flatten()\n",
    "\n",
    "    # use the preceding operators to construct networks\n",
    "    def construct(self, x):\n",
    "        x = self.max_pool2d(self.relu(self.conv1(x)))\n",
    "        x = self.max_pool2d(self.relu(self.conv2(x)))\n",
    "        x = self.flatten(x)\n",
    "        x = self.relu(self.fc1(x))\n",
    "        x = self.relu(self.fc2(x))\n",
    "        x = self.fc3(x)\n",
    "        return x"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 定义回调函数EvalCallBack"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "实现思想：每隔n个epoch验证一次模型精度，需要在自定义回调函数中实现，如需了解自定义回调函数的详细用法，请参考[API说明](https://www.mindspore.cn/doc/api_python/zh-CN/master/mindspore/mindspore.train.html#mindspore.train.callback.Callback)。\n",
    "\n",
    "核心实现：回调函数的`epoch_end`内设置验证点，如下：\n",
    "\n",
    "`cur_epoch % eval_per_epoch == 0`：即每`eval_per_epoch`个epoch结束时，验证一次模型精度。\n",
    "\n",
    "- `cur_epoch`：当前训练过程的`epoch`数值。\n",
    "- `eval_per_epoch`：用户自定义数值，即验证频次。\n",
    "\n",
    "其他参数解释：\n",
    "\n",
    "- `model`：MindSpore中的`Model`类。\n",
    "- `eval_dataset`：验证数据集。\n",
    "- `epoch_per_eval`：记录验证模型的精度和相应的epoch数，其数据形式为`{\"epoch\":[],\"acc\":[]}`。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mindspore.train.callback import Callback\n",
    "\n",
    "class EvalCallBack(Callback):\n",
    "    def __init__(self, model, eval_dataset, eval_per_epoch, epoch_per_eval):\n",
    "        self.model = model\n",
    "        self.eval_dataset = eval_dataset\n",
    "        self.eval_per_epoch = eval_per_epoch\n",
    "        self.epoch_per_eval = epoch_per_eval\n",
    "        \n",
    "    def epoch_end(self, run_context):\n",
    "        cb_param = run_context.original_args()\n",
    "        cur_epoch = cb_param.cur_epoch_num\n",
    "        if cur_epoch % self.eval_per_epoch == 0:\n",
    "            acc = self.model.eval(self.eval_dataset, dataset_sink_mode=False)\n",
    "            self.epoch_per_eval[\"epoch\"].append(cur_epoch)\n",
    "            self.epoch_per_eval[\"acc\"].append(acc[\"Accuracy\"])\n",
    "            print(acc)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 定义训练网络并执行"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在保存模型的参数`CheckpointConfig`中，需计算好单个`epoch`中的`step`数，根据保存模型参数`ckpt`文件时，需要间隔的`step`数来设置，本次示例每个`epoch`有1875个`step`，按照每两个`epoch`验证一次的思想，这里设置`save_checkpoint_steps=eval_per_epoch*1875`，\n",
    "其中变量`eval_per_epoch`等于2。\n",
    "\n",
    "参数解释：\n",
    "\n",
    "- `train_data_path`：训练数据集地址。\n",
    "- `eval_data_path`：验证数据集地址。\n",
    "- `train_data`：训练数据集。\n",
    "- `eval_data`：验证数据集。\n",
    "- `net_loss`：定义损失函数。\n",
    "- `net-opt`：定义优化器函数。\n",
    "- `config_ck`：配置保存模型信息。\n",
    "    - `save_checkpoint_steps`：每多少个step保存一次模型权重参数`ckpt`文件。\n",
    "    - `keep_checkpoint_max`：设置保存模型的权重参数`cpkt`文件的数量上限。\n",
    "- `ckpoint_cb`：配置模型权重参数`ckpt`文件保存名称的前缀信息及保存路径信息。\n",
    "- `model`：MindSpore中的`Model`类。\n",
    "- `model.train`：`Model`类的执行训练函数。\n",
    "- `epoch_per_eval`：定义收集`epoch`数和对应模型精度信息的字典。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "epoch: 1 step: 375, loss is 2.3327153\n",
      "epoch: 1 step: 750, loss is 2.301087\n",
      "epoch: 1 step: 1125, loss is 0.18899053\n",
      "epoch: 1 step: 1500, loss is 0.31486228\n",
      "epoch: 1 step: 1875, loss is 0.14021991\n",
      "epoch: 2 step: 375, loss is 0.049191322\n",
      "epoch: 2 step: 750, loss is 0.08493232\n",
      "epoch: 2 step: 1125, loss is 0.2740858\n",
      "epoch: 2 step: 1500, loss is 0.0712947\n",
      "epoch: 2 step: 1875, loss is 0.084480055\n",
      "{'Accuracy': 0.9782652243589743}\n",
      "epoch: 3 step: 375, loss is 0.056499712\n",
      "epoch: 3 step: 750, loss is 0.10981669\n",
      "epoch: 3 step: 1125, loss is 0.013717058\n",
      "epoch: 3 step: 1500, loss is 0.16365167\n",
      "epoch: 3 step: 1875, loss is 0.052067317\n",
      "epoch: 4 step: 375, loss is 0.05080418\n",
      "epoch: 4 step: 750, loss is 0.013522813\n",
      "epoch: 4 step: 1125, loss is 0.08582015\n",
      "epoch: 4 step: 1500, loss is 0.04939629\n",
      "epoch: 4 step: 1875, loss is 0.09115914\n",
      "{'Accuracy': 0.9836738782051282}\n",
      "epoch: 5 step: 375, loss is 0.0035727315\n",
      "epoch: 5 step: 750, loss is 0.03130674\n",
      "epoch: 5 step: 1125, loss is 0.0011531024\n",
      "epoch: 5 step: 1500, loss is 0.009147665\n",
      "epoch: 5 step: 1875, loss is 0.0024722838\n",
      "epoch: 6 step: 375, loss is 0.03595736\n",
      "epoch: 6 step: 750, loss is 0.004377359\n",
      "epoch: 6 step: 1125, loss is 0.044095017\n",
      "epoch: 6 step: 1500, loss is 0.016356776\n",
      "epoch: 6 step: 1875, loss is 0.01198354\n",
      "{'Accuracy': 0.9818709935897436}\n",
      "epoch: 7 step: 375, loss is 0.011158295\n",
      "epoch: 7 step: 750, loss is 0.021831619\n",
      "epoch: 7 step: 1125, loss is 0.0027707873\n",
      "epoch: 7 step: 1500, loss is 0.0001371978\n",
      "epoch: 7 step: 1875, loss is 0.00040429938\n",
      "epoch: 8 step: 375, loss is 0.005541572\n",
      "epoch: 8 step: 750, loss is 0.0038450873\n",
      "epoch: 8 step: 1125, loss is 0.1304332\n",
      "epoch: 8 step: 1500, loss is 0.021286076\n",
      "epoch: 8 step: 1875, loss is 0.025266083\n",
      "{'Accuracy': 0.9817708333333334}\n",
      "epoch: 9 step: 375, loss is 0.0045793867\n",
      "epoch: 9 step: 750, loss is 0.009571521\n",
      "epoch: 9 step: 1125, loss is 0.06868767\n",
      "epoch: 9 step: 1500, loss is 0.00035104403\n",
      "epoch: 9 step: 1875, loss is 0.0010347537\n",
      "epoch: 10 step: 375, loss is 0.058423545\n",
      "epoch: 10 step: 750, loss is 0.0044561117\n",
      "epoch: 10 step: 1125, loss is 2.982349e-05\n",
      "epoch: 10 step: 1500, loss is 0.040188752\n",
      "epoch: 10 step: 1875, loss is 0.047129657\n",
      "{'Accuracy': 0.9833733974358975}\n"
     ]
    }
   ],
   "source": [
    "from mindspore.train.callback import ModelCheckpoint, CheckpointConfig, LossMonitor\n",
    "from mindspore import context, Model\n",
    "from mindspore.nn.metrics import Accuracy\n",
    "from mindspore.nn.loss import SoftmaxCrossEntropyWithLogits\n",
    "import os\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    context.set_context(mode=context.GRAPH_MODE, device_target=\"CPU\")\n",
    "    train_data_path = \"./datasets/MNIST_Data/train\"\n",
    "    eval_data_path = \"./datasets/MNIST_Data/test\"\n",
    "    model_path = \"./models/ckpt/mindspore_evaluate_the_model_during_training/\"\n",
    "    \n",
    "    # clean up old run files before in Linux\n",
    "    os.system('rm -f {}*.ckpt {}*.meta {}*.pb'.format(model_path, model_path, model_path))\n",
    "    \n",
    "    epoch_size = 10\n",
    "    eval_per_epoch = 2\n",
    "    repeat_size = 1\n",
    "    network = LeNet5()\n",
    "    train_data = create_dataset(train_data_path, repeat_size=repeat_size)\n",
    "    eval_data = create_dataset(eval_data_path, repeat_size=repeat_size)\n",
    "    \n",
    "    # define the loss function\n",
    "    net_loss = SoftmaxCrossEntropyWithLogits(sparse=True, reduction='mean')\n",
    "    # define the optimizer\n",
    "    net_opt = nn.Momentum(network.trainable_params(), learning_rate=0.01, momentum=0.9)\n",
    "    config_ck = CheckpointConfig(save_checkpoint_steps=eval_per_epoch*1875, keep_checkpoint_max=15)\n",
    "    ckpoint_cb = ModelCheckpoint(prefix=\"checkpoint_lenet\", directory=model_path, config=config_ck)\n",
    "    model = Model(network, net_loss, net_opt, metrics={\"Accuracy\": Accuracy()})\n",
    "    \n",
    "    epoch_per_eval = {\"epoch\": [], \"acc\": []}\n",
    "    eval_cb = EvalCallBack(model, eval_data, eval_per_epoch, epoch_per_eval)\n",
    "    \n",
    "    model.train(epoch_size, train_data, callbacks=[ckpoint_cb, LossMonitor(375), eval_cb],\n",
    "                dataset_sink_mode=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "在同一目录的文件夹中可以看到`lenet_ckpt`文件夹中，保存了5个模型，和一个计算图相关数据，其结构如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "./models/ckpt/mindspore_evaluate_the_model_during_training\n",
      "├── checkpoint_lenet-10_1875.ckpt\n",
      "├── checkpoint_lenet-2_1875.ckpt\n",
      "├── checkpoint_lenet-4_1875.ckpt\n",
      "├── checkpoint_lenet-6_1875.ckpt\n",
      "├── checkpoint_lenet-8_1875.ckpt\n",
      "└── checkpoint_lenet-graph.meta\n",
      "\n",
      "0 directories, 6 files\n"
     ]
    }
   ],
   "source": [
    "!tree ./models/ckpt/mindspore_evaluate_the_model_during_training"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 绘制不同epoch下模型的精度"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "定义绘图函数`eval_show`，将`epoch_per_eval`载入到`eval_show`中，绘制出不同`epoch`下模型的验证精度折线图。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEWCAYAAABxMXBSAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAx4klEQVR4nO3debyc8/n/8dc7EgSxJeGHaGKrip2IfWkUobXFFntr60LLt9WW0m9bLW1Vq/WlizqHIPY1FEEklNpOIkHEEkHWEktEBJHk+v3xuYfJcXLOTDJz7nPOvJ+Px3lk5t7mmpxkrvks9/VRRGBmZlaqTnkHYGZm7YsTh5mZlcWJw8zMyuLEYWZmZXHiMDOzsjhxmJlZWZw4rOok9ZEUkjqXcOw3JT3aGnHVCklHS7p/Kc6/V9LxlYyphdcr+d+L5cOJwxYh6XVJ8yT1aLT9mew/c5+cQrMlFBFDI2LvUo6V9EtJ1zY6f9+IGFKd6FpH9m93w7zj6CicOKwprwFHFp5I2hxYIb9w2ob2+A24PcZcSbX+/qvFicOacg1wXNHz44Griw+QtIqkqyXNlPSGpHMldcr2LSPpIklvS5oEfL2Jc+skzZA0TdJvJC1TSmCSbpb0X0nvS3pE0qZF+7pK+mMWz/uSHpXUNdu3i6T/SJolaYqkb2bbR0k6qegai3SVZd9UT5X0CvBKtu0v2TVmSxotadei45eR9DNJr0r6INu/rqTLJP2x0XsZJul/mniPf5N0UaNtd0r6Yfb4rKLrvyDp4EbxPybpYknvAL9s4j01Gb+kgcDPgCMkzZE0rvHfkaRO2e/6DUlvZf8GVsn2FbqYjpc0Ofv9n9PM73Kxv6/M0U1dR1J/SY9nv8sZki6VtOzifmeSHsl2jcve1xGLi8lKFBH+8c9nP8DrwNeAl4BNgGWAqUBvIIA+2XFXA3cC3YA+wMvAidm+7wAvAusCqwMjs3M7Z/tvB/4BrAisATwFfDvb903g0WbiOyF7zeWAPwNji/ZdBowC1sni3ik7rjfwAakV1QXoDmyVnTMKOKnoGou8fhb3A9n76JptOya7RmfgR8B/geWzfT8GngM2BgRsmR3bH5gOdMqO6wHMBdZs4j3uBkwBlD1fDfgIWDt7fhiwNumL3xHAh8BaRfHPB76fxde1iffUXPy/BK5tFM9nf0fZ3/9EYH1gJeA24JpsX5/s7+uf2etuCXwCbLKY3+Xifl/NXgfYFtghi78PMAE4o4XfWQAb5v3/q6P85B6Af9rWD58njnOB3wIDs/+EnbP/fH2y/+TzgL5F530bGJU9fgj4TtG+vbNzOwNrZh8CXYv2HwmMzB4v8iHXQqyrZtddJfsQ/QjYsonjzgZuX8w1PvtQbOr1s+sPaCGO9wqvS0q4By7muAnAXtnj04B7FnOcgMnAbtnzk4GHmnn9sYXXzOKf3Gh/s3+njeL/Jc0njhHA94r2bQx8WvQhHkCvov1PAYObeM3mfl8lXyfbd0bx77ep3xlOHBX9cVeVLc41wFGkD52rG+3rQfrm/kbRtjdI3xwhfRue0mhfQe/s3BlZV8MsUutjjZYCyrqBfpd108wmJblCPD2A5YFXmzh13cVsL1Xxe0HSmZImZN0rs0iJqzCZoLnXGkL6tk/25zVNHRTpk+4GPh9nOgoYWvT6x0kaW/T3t1nR638h3sZaiL8la/PF33vhC0HBf4sezyW1TBpr7vfV7HUkfVnS3VmX5Wzggibib/bvwJaOE4c1KSLeIA2S70fqjij2NulbZu+ibV8CpmWPZ5A+QIv3FUwhtTh6RMSq2c/KEbEpLTsKOJDUIlqF9M0U0jf0t4GPgQ2aOG/KYrZD6uYpHvj/f00c81kJ6Ww84CfA4cBqEbEq8H4WQ0uvdS1woKQtSd2AdyzmOIDrgUMl9Qa2B27NXr83qQvnNKB79vrPF73+IvE2VkL8LZXLns4Xf+/zgTdbOK+x5n5fLfkbqSt0o4hYmTQuo0bHuOx3FTlxWHNOJDX5PyzeGBELgJuA8yV1yz7Mfkj6YCTb9wNJvSStBpxVdO4M4H7gj5JWzgZbN5C0ewnxdCMlnXdIH/YXFF13IVAP/EnS2lnrZEdJy5G+rX9N0uGSOkvqLmmr7NSxwCBJKyhN1zyxhBjmAzOBzpL+F1i5aP8VwK8lbaRkC0ndsxinAk+TWhq3RsRHi3uRiHiG9OF6BTA8ImZlu1YkfSjOBJD0LVKLo1Qtxf8m0EfZRIcmXA/8j6T1JK1E+h3cGBHzy4ihpd9XKe9hNjBH0leA75ZwzpukcRmrACcOW6yIeDUiGhaz+/ukb+uTgEeB60gfBJC+EQ8HxgFj+GKL5ThgWeAFUv/6LcBaJYR0NalrZFp27hON9p9JGph+GngX+D1pMHoyqeX0o2z7WNKAK8DFpPGaN0ldSUNp3nDgPtJkgDdI35qLu0X+REqc95M+3OpIA7wFQ4DNWUw3VSPXkVpX1xU2RMQLwB+Bx7OYNwceK+FapcZ/c/bnO5LGNHF+fRb7I6QW6cekfwtLosnfV4nnHUWa8PBP4MYSzvklMCTr3jt8iaK1zxRmbZhZK5C0G6ll1jv8n8/aKbc4zFqJpC7A6cAVThrWnjlxmLUCSZsAs0hdcn/ONRizpeSuKjMzK4tbHGZmVpaaKADWo0eP6NOnT95hmJm1K6NHj347Ino23l4TiaNPnz40NCxuVqmZmTVF0htNbXdXlZmZlcWJw8zMyuLEYWZmZXHiMDOzsjhxmJlZWZw4zMysLE4cZmZWFicOq5wXX4Q77sg7CjOrMicOq5wTToCDD4bTT4cFC/KOxsyqxInDKmPCBHj8cdhiC7jkEhg0CD78sOXzzKzdceKwyqivh86d4YEH4NJL4e67Yffd4b//zTsyM6swJw5bep9+CldfDfvvD2usAaeeCnfemVohO+wA48fnHaGZVZAThy29f/0L3norjXEUfOMb8Mgj8MknsPPO8NBD+cVnZhXlxGFLr74e1loLBg5cdPu228KTT0KvXrDPPjBkSD7xmVlFOXHY0pkxA+65B44/Po1xNPalL8Fjj8Eee8A3vwn/+7/gVSfN2jUnDls6Q4akqbfF3VSNrbJKSi4nnAC//jUcd1zqwjKzdqkmFnKyKolI3VS77gobbdT8sV26wBVXwPrrw7nnwpQpcPvtsNpqrROrmVWMWxy25B59FF55BU48sbTjJTjnHBg6NN3zseOOMGlSdWM0s4pz4rAlV18P3brBoYeWd95RR6X7Pd56K03XffLJ6sRnVsumTElT4+fNq/ilnThsycyeDTfdBIMHw4orln/+brulVke3bmng/LbbKh6iWc164AHYZpt0f9Xzz1f88k4ctmRuugnmzm1+ULwlG28MTzwBW22VWi0XX+wZV2ZLY+HCNAFln31gzTWhoSElkApz4rAlU1cHffvC9tsv3XV69kw3Bw4aBD/8IXz/+zB/fmViNKsl77wDX/96mvJ+9NGpC3jjjavyUk4cVr4XXkgthRNOSAPeS6tr19SCOfNMuOyyVGF3zpylv65ZrXj66dSyeOgh+NvfUhfVknQhl8iJw8pXKGh47LGVu2anTvCHP8Bf/5ru+dhtN5g+vXLXN+uIIlKi2GWX9CXu0UfhO9+pzBe6ZjhxWHkKBQ0POCAVNKy0734X7roLXn45zbh67rnKv4ZZR/Dhh+nL2/e+B3vuCaNHw3bbtcpLO3FYee6+G2bOXLpB8Zbstx/8+99prGOXXdIMETP73EsvpfHF665Lg+F33w3du7fayztxWHnq62HttdOsjWraeus0uNe7d0ok9fXVfT2z9uLmm6FfP3jzTRg+PFVi6NS6H+VOHFa66dObL2hYaeuum/psBwxId6efe66n61rtmjcPzjgDDj8cNtsMxoyBvfbKJRQnDivdkCFpnng1u6kaW3nl1Aw/6SQ4/3w45hgXSLTaM3VqulH2L3+BH/wAHn44fbHKiYscWmkKBQ132w023LB1X7tLF7j8cthgAzj77M8LJLZin65Zbh58EI48Ej7+GG64AY44Iu+IqtvikDRQ0kuSJko6q4n9vSWNkPSspFGSehXtu1DSeEkTJF0ipfllku6TNC7b93dJy1TzPVjm3/+GiRNLL2hYaRKcdRZcf30a+9hpJ3j11XxiMWsNCxfCb34De++dZjA+/XSbSBpQxcSRfaBfBuwL9AWOlNS30WEXAVdHxBbAecBvs3N3AnYGtgA2A7YDds/OOTwitsy29wQOq9Z7sCKFgoaHHJJvHIMHw4gR8Pbbabru44/nG49ZNbz7Luy/P/z856m18eST8JWv5B3VZ6rZ4ugPTIyISRExD7gBOLDRMX2BwmLUI4v2B7A8sCywHNAFeBMgImZnx3TO9nu0tNpmz04zOY48sqp3o5Zsl13SneurrJIGzm+5Je+IzCqnUF/qgQdSJYVrr4WVVso7qkVUM3GsA0wpej4121ZsHDAoe3ww0E1S94h4nJRIZmQ/wyNiQuEkScOBt4APgCY/NSSdIqlBUsPMmTMr8X5q1403Ln1Bw0rbaKPU2thmGzjsMLjoIs+4svYtAv7xD9h55/T40UfTzX1Vvgt8SeQ9q+pMYHdJz5C6oqYBCyRtCGwC9CIlmwGSdi2cFBH7AGuRWiMDmrpwRFweEf0iol/Pnj2r/DY6uLo62HRT6N8/70gW1bNn6rY67DD48Y/T2gMukGjt0Ycfpmnu3/lOakWPGdP2/r8VqWbimAYUzxfrlW37TERMj4hBEbE1cE62bRap9fFERMyJiDnAvcCOjc79GLiTL3Z/WSWNH5/6V088sU1+82H55dNMk5/+NNXsOfBA+OCDvKMyK91LL6XxumuvhV/9Cv71rzY/Y7CaieNpYCNJ60laFhgMDCs+QFIPSYUYzgYKtwdPJrVEOkvqQmqNTJC0kqS1snM7A18HXqzie7D6+jQd9phj8o5k8Tp1gt/9LjXzhw9PU4anTWv5PLO83XJLqi81Ywbce28qid7Kd4EviapFGBHzgdOA4cAE4KaIGC/pPEkHZIftAbwk6WVgTeD8bPstwKvAc6RxkHERcRewIjBM0rPAWNI4x9+r9R5q3rx5cM01qaBhe+juO+WUdLPgxInpG9yzz+YdkVnTPv00rT9z2GFpXZtnnql+GZ8KUtTAgGK/fv2ioaEh7zDan9tuS9Nv//WvVC+qvRg3Li1oU5gN1o7+Q1oNmDYt3Y/x2GNp4bKLLoJll807qiZJGh0R/Rpvb/ttIstPfT2ss077++Ddcss0XXf99VMC+ec/847ILHnooTQTcOzYdDPrJZe02aTRHCcOa9q0aanP9fjjYZl2eHN+r17pbve99kpdWGefne7ENcvDwoVwwQXp32P37uku8MGD845qiTlxWNMKBQ2/9a28I1ly3bqlRaG+/e00eH7UUanej1lreu+9NE54zjmpsu1TT8Emm+Qd1VJxkUP7okJBw913b/2ChpXWuXOaprv++mnK7tSpcMcd0KNH3pFZLRg9Gg49NLXg/+//0r1GbXFae5nc4rAveuSRVEAwr4KGlSbBT36S7oBvaIAdd4RXXsk7KuvIIlJF5512ggUL0v+p007rEEkDnDisKfX1aR2MvAsaVtrhh6fByffeS8njscfyjsg6orlz4ZvfTF2ke+yR7gLfYYe8o6ooJw5b1Pvvf17QcIUV8o6m8nbaKc24Wn112HNPuOmmvCOyjuSVV1KSuOYa+MUv0oqZHbBb1InDFnXjjfDRR22roGGlbbhhKpC43XZpPv3vf+8Cibb0brsNtt02jWfccw/88pftc0ZiCZw4bFF1dWk94+22yzuS6urePZWtHjw4LRD1ne+4QKItmU8/hR/9KHXtbrJJugt84MC8o6oqz6qyzz3/fJoqePHFHWYQr1nLLw9Dh6YZVxdcAG+8kbquVl4578isvZg+PbVaH300zZj64x9hueXyjqrq3OKwz7WHgoaV1qkTnH9+urv8wQdh113TlF2zlowcCVtvnQa/hw6FSy+tiaQBThxWUChoeOCBHXIwr0UnnZT6pV97DbbfPpWEMGvKwoXphtKvfQ1WWy210o86Ku+oWpUThyV33ZXW8e7Ig+It2Xvv1OXQqVNqedx7b94RWVvz3ntw0EGphM1hh6XSIZtumndUrc6Jw5K6ulTQcO+9844kX1tskRau2mgj2H//tMaHGaRB7223TV8oLrkkFSns1i3vqHLhxGGpT3/48HTTUgedPliWtddOd/rus0+abfXTn7pAYi2LgCuuSDeNfvpp+rfx/e/XxgSSxXDisI5R0LDSVloJ7rwTvvtduPDCNG33o4/yjspa29y5qfv25JNT9+WYMSmB1Dgnjlq3cGGaTbXHHrDBBnlH07Z07gyXXZYW2rn55nSn+cyZeUdlrWXixJQkrroKfv5zuO++9rESZitw4qh1jzwCkyZ1nIKGlSalm7tuvjn1ce+4I7z8ct5RWbXdfnsaz5g6Nc22O+88d+MWceKodYWChoMG5R1J23booWne/uzZKXn8+995R2TVMH8+/PjH6f/Dl7+cuqb23TfvqNocJ45a9v77cMstaQ56RyxoWGk77JBqXPXokebwX3993hFZJc2YAQMGpK7J7343Tc3u3TvvqNokJ45adsMNacDX3VSl22CDlDy23z4l3AsucIHEjuDhh9Nd4KNHw7XXwl//WjN3gS8JJ45aVlcHm2+e+nKtdKuvngokHnVUWg705JPTNE1rfyJSdeQBA2DVVdM9PEcfnXdUbZ6LHNaq555Ld73++c81PR99iS23XPpmuv768JvfwOTJaQB9lVXyjsxKNWsWHH88DBuW7gKvq6vZG/rK5RZHrSoUNPS3qyUnwa9/nT5wRo6EXXZJCcTavrFjU0v7nnvSl6cbb3TSKIMTRy0qFDQ86KDaLGhYaSeckMpQTJ6cBtDHjMk7ImtOXV36PX3ySRrbOP10t7rL5MRRi4YNg3feqe2ChpX2ta+lNcw7d4bddoN//SvviKyxwkSQk05KrcMxY9JSwlY2J45aVFcHvXrBXnvlHUnHstlmaXB1443hgAPSzBxrG159NSWJ+no499xUm22NNfKOqt2qauKQNFDSS5ImSjqrif29JY2Q9KykUZJ6Fe27UNJ4SRMkXaJkBUn/kvRitu931Yy/Q5oyxQUNq2mttVL3x377pRXhzjzTBRLzduedaTzjjTfg7rvTuJT/7S+VqiUOScsAlwH7An2BIyX1bXTYRcDVEbEFcB7w2+zcnYCdgS2AzYDtgN0L50TEV4CtgZ0l+bbOcgwZkqYguqBh9ay0EtxxB5x2WlpK9LDDUrE8a13z56fKxgcdBBtumLqmvv71vKPqEKrZ4ugPTIyISRExD7gBOLDRMX2Bh7LHI4v2B7A8sCywHNAFeDMi5kbESIDsmmOAXlhpFi6EK6+Er341TSO16llmmbRmw8UXp7pHAwbAW2/lHVXt+O9/07jThRfCt7+d7gLv0yfvqDqMaiaOdYApRc+nZtuKjQMKRZIOBrpJ6h4Rj5MSyYzsZ3hETCg+UdKqwP7AiKZeXNIpkhokNcx0RdPk4Ydd0LA1SXDGGXDrrfDss2kmz4sv5h1Vx/fII+ku8Keegquvhr//HZZfPu+oOpQWE4ek0ZJOlbRaFV7/TGB3Sc+QuqKmAQskbQhsQmpNrAMMkLRrUUydgeuBSyJiUlMXjojLI6JfRPTr6VLISX19ukHNBQ1b18EHw6hR8OGHaYD2kUfyjqhjioA//CG17rp1SxMVjj0276g6pFJaHEcAawNPS7pB0j5SSZOepwHrFj3vlW37TERMj4hBEbE1cE62bRap9fFERMyJiDnAvUDx6imXA69ExJ9LiMMg3SVbKGjYtWve0dSe/v3hiSdgzTXTbLahQ/OOqGN5//30hegnP0ljGg0NqZyOVUWLiSMiJkbEOcCXgeuAeuANSb+StHozpz4NbCRpPUnLAoOBYcUHSOohqRDD2dm1ASaTWiKdJXUhtUYmZOf8BlgFOKPE92iQChp+/LG7qfK03nrwn/+kVscxx6RSJS6QuPTGjUuzpu6+G/70p1T6ZeWV846qQyupVpWkLYBvAfsBtwJDgV1IA9tbNXVORMyXdBowHFgGqI+I8ZLOAxoiYhiwB/BbSQE8ApyanX4LMAB4jjRQfl9E3JVN1z0HeBEYkzV8Lo2IK8p837Wnrg622AK22SbvSGrbaqul6dAnnZRWlXvsMfjKV1LXykorpZ/FPS48X3ZZ3+lccNVVqQT66qun7sCdd847opqgaOEbj6TRwCygDrg1Ij4p2ndbRLT5DvN+/fpFQ0ND3mHk59lnYcst4S9/gR/8IO9oDFJL4/zz4W9/gw8+gDlzSm99dO7cfIJpKfE09bhLl+q+30r76KP0b/mKK9KYxnXXpW5AqyhJoyOi3xe2l5A41l/cAHR7UfOJ44wz0gfU9OnQvXve0VhTItKHYSGJzJmzdI/nzEmD8aVabrnyk01zCWqllap3k92kSWlFxmeegZ/9zMu6VtHiEkcpXVUnSbowG7Qmm131o4g4t8IxWjV88kkq/33QQU4abZmUVmFcYYXKfXNesCDdeLikieeDD+DNNxfd9/HHpb9+166VaQ0VHq+wQhrHOO649Pd1113wjW9U5u/KylJK4tg3In5WeBIR70naD3DiaA9c0LB2LbNM+tDt1i2VQqmE+fO/mFyaSzyNH8+alcreFG8vdREsKbXMttkmzRBcb73KvCcrWymJYxlJyxXGNiR1Jd3Nbe1BXR2su266i9ZsaXXunFbKW3XVyl1z3rzSE8/KK6caYL6hL1elJI6hwAhJV2bPvwUMqV5IVjFTpsD996dqoO4DtrZq2WXTrKjVm5vdb21Ji4kjIn4v6Vlgz2zTryNieHXDsoq46ioXNDSziivpPo6IuJd097a1F4WChgMGuC/YzCqqlFpVO0h6WtIcSfMkLZA0uzWCs6UwahS89prvFDeziiulVtWlwJHAK0BX4CTSOhvWltXXpwHMgw/OOxIz62BKKqseEROBZSJiQURcCQysbli2VGbNSqW8XdDQzKqglDGOuVmRwrGSLiStj+G1ytuy6693QUMzq5pSEsCx2XGnAR+SSqUfUs2gbCnV1aXaVFtvnXckZtYBNdviyNYNvyAijgY+Bn7VKlHZkhs3DkaPTsuWuoKqmVVBsy2OiFgA9M66qqw9qK9PN1QddVTekZhZB1XKGMck4DFJw0hdVQBExJ+qFpUtmUJBw4MPdkFDM6uaUhLHq9lPJ6BbdcOxpXLnnfDuuy5oaGZVVUrJEY9rtBd1dfClL8Gee7Z8rJnZEmoxcUgaSVq+dRERMaAqEdmSmTwZHnggLUfqgoZmVkWldFWdWfR4edJU3PnVCceWmAsamlkrKaWranSjTY9JeqpK8diSKBQ03HNP6NMn72jMrIMrpauquEh+J2BbYJWqRWTlGzkSXn8dLrgg70jMrAaU0lU1mjTGIVIX1WuAa1m0JS5oaGatqJSuKi/m0Ja9914qaHjSSV5O08xaRSnrcZwqadWi56tJ+l5Vo7LSXX99uvHPBQ3NrJWUUuTw5IiYVXgSEe8BJ1ctIitPXR1stZULGppZqyklcSwjfV4tLyt86NpVbcHYsTBmjFsbZtaqShkcvw+4UdI/suffzrZZ3urrYbnlXNDQzFpVKYnjp8ApwHez5w8AV1QtIivNxx9/XtBw9dVbPt7MrEJK6arqCvwzIg6NiENJSWO5Ui4uaaCklyRNlHRWE/t7Sxoh6VlJoyT1Ktp3oaTxkiZIuqTQXSbpfElTJM0p7S12UHfemWZUuaChmbWyUhLHCFLyKOgKPNjSSdlYyGXAvkBf4EhJfRsddhFwdURsAZwH/DY7dydgZ2ALYDNgO2D37Jy7gP4lxN2xuaChmeWklMSxfER89u0+e7xCCef1ByZGxKSImAfcABzY6Ji+wEPZ45FF+4NUF2tZUuumC/Bm9vpPRMSMEl6/43rjDXjwwVSXqpOXfzez1lXKp86HkrYpPJG0LfBRCeetA0wpej4121ZsHDAoe3ww0E1S94h4nJRIZmQ/wyNiQgmv+RlJp0hqkNQwc+bMck5t+666Kv3pgoZmloNSEscZwM2S/i3pUeBG4LQKvf6ZwO6SniF1RU0DFkjaENgE6EVKNgMk7VrOhSPi8ojoFxH9evbsWaFw24BCQcOvfQ169847GjOrQaWUHHla0leAjbNNL0XEpyVcexqwbtHzXtm24mtPJ2txSFoJOCQiZkk6GXii0EUm6V5gR+DfJbxux/bQQ6mr6ne/yzsSM6tRpXaQb0waj9iGNMh9XAnnPA1sJGk9ScsCg4FhxQdI6iGpEMPZQH32eDKpJdJZUhdSa6SsrqoOq74eVlsNDjoo70jMrEaVUqvqF8D/ZT9fBS4EDmjpvIiYT+rSGk760L8pIsZLOk9S4fw9gJckvQysCZyfbb+FtM75c6RxkHERcVcWz4WSpgIrSJoq6Zclvtf277334Lbb4OijXdDQzHKjiC+sCrvoAdJzwJbAMxGxpaQ1gWsjYq/WCLAS+vXrFw0NDXmHsfQuuwxOOw2eeSbVpzIzqyJJoyOiX+PtpXRVfRQRC4H5klYG3mLRsQtrLXV1qZihk4aZ5aiUkiMNWVn1f5IWdZoDPF7NoKwJzzyTfi69NO9IzKzGlTKrqrD2xt8l3QesHBHPVjcs+wIXNDSzNqKUFsdnIuL1KsVhzfn4Yxg6FAYNSjOqzMxy5HoV7cEdd7igoZm1GU4c7UFdXbpLfMCAvCMxM1t8V5WkZhd5iIh3Kx+OfcHrr8OIEfCLX7igoZm1Cc2NcYwmValVE/sCWL8qEdmiXNDQzNqYxSaOiFivNQOxJhQKGu61V1p7w8ysDSil5IgkHSPp59nzL0nyQkqtYcQImDzZg+Jm1qaU0mn+V1Jl2sINBB+QVvazaquvT+uJu6ChmbUhpSSO7SPiVOBjgIh4j7Qyn1XTu+/C7bengobLlbTEu5lZqyglcXyarR8eAJJ6AgurGpXBddfBJ5/AiSfmHYmZ2SJKSRyXALcDa0g6H3gUuKCqUVm6d2ObbWDLLfOOxMxsEaXUqhoqaTSwJ2lq7kHlrv9tZRozBsaOTWXUzczamFJvAHwLuL54n28ArKJCQcMjj8w7EjOzLyj1BsAvAe9lj1clLe3q+zyq4aOPUkHDQw5xQUMza5MWO8YREetFxPrAg8D+EdEjIroD3wDub60Aa84dd8CsWb53w8zarFIGx3eIiHsKTyLiXmCn6oVU4+rqYL314KtfzTsSM7MmlZI4pks6V1Kf7OccYHq1A6tJhYKG3/qWCxqaWZtVyqfTkUBP0pTc24E1sm1WaVdeCRIcf3zekZiZLVYp03HfBU6X1C09jTnVD6sGLViQEsfee7ugoZm1aaUUOdxc0jPA88B4SaMlbVb90GrMiBEwZYoHxc2szSulq+ofwA8jondE9AZ+BFxe3bBqUKGg4YEH5h2JmVmzSkkcK0bEyMKTiBgFrFi1iGrRO++kgobHHOOChmbW5rU4xgFMytbiuCZ7fgwwqXoh1aDrroN589xNZWbtQiktjhNIs6puy356ZtusEiLSvRvbbuuChmbWLrSYOCLivYj4QURsk/2cnq3J0SJJAyW9JGmipLOa2N9b0ghJz0oaJalX0b4LJY2XNEHSJZKUbd9W0nPZNT/b3m6NGQPjxrl8upm1G80VORzW3IkRcUBz+7M1PC4D9gKmAk9LGhYRLxQddhFwdUQMkTQA+C1wrKSdgJ2BLbLjHgV2B0YBfwNOBp4E7gEGAvc2F0ubVl8Pyy/vgoZm1m40N8axIzCFVBX3SVKBw3L0ByZGxCQASTcABwLFiaMv8MPs8UjgjuxxAMuTVhoU0AV4U9JawMoR8UR2zauBg2iviaO4oOGqq+YdjZlZSZrrqvp/wM+AzYC/kFoOb0fEwxHxcAnXXoeUeAqmZtuKjQMGZY8PBrpJ6h4Rj5MSyYzsZ3i2Bsg62XWau2b7cfvt8P77HhQ3s3alueq4CyLivog4HtgBmAiMknRaBV//TGD37AbD3YFpwAJJGwKbAL1IiWGApF3LubCkUyQ1SGqYOXNmBUOuoEJBwz32yDsSM7OSNTs4Lmk5SYOAa4FT+XwZ2VJMA9Ytet4r2/aZiJgeEYMiYmvgnGzbLFLr44mImJOVOLmX1HU2LbvOYq9ZdO3LI6JfRPTr2bNniSG3otdeg4ceSq0NFzQ0s3ZksZ9Y2fjB48A2wK8iYruI+HVENPlB3YSngY0krSdpWWAwsMiAu6QekgoxnA3UZ48nk1oinSV1IbVGJkTEDGC2pB2y2VTHAXeWGE/b4oKGZtZONfdV9xhgI+B04D+SZmc/H0ia3dKFI2I+cBowHJgA3BQR4yWdJ6kwI2sP4CVJLwNrAudn228BXgWeI42DjIuIu7J93wOuIHWdvUp7HBhfsACuugr22QfWXbfFw83M2hJFRN4xVF2/fv2ioaEh7zA+N3w4DBwIN90Ehx2WdzRmZk2SNDoi+jXe7s71PNTXQ/fucECzt8KYmbVJThyt7Z130rriLmhoZu2UE0dru/baVNDQJUbMrJ1y4mhNhYKG/frB5pvnHY2Z2RJx4mhNo0fDc8+5tWFm7ZoTR2sqFDQcPDjvSMzMlpgTR2v56KO0YNOhh7qgoZm1a04creW221JBQ3dTmVk758TRWurqYP31Ybfd8o7EzGypOHG0hkmTYORIFzQ0sw7Bn2Kt4corU8JwQUMz6wCcOKqtuKBhr14tHm5m1tY5cVTbAw/A1Kle5c/MOgwnjmqrr4cePVzQ0Mw6DCeOanr77c8LGi67bN7RmJlVhBNHNV17LXz6qe/dMLMOxYmjWiJSN9V228Fmm+UdjZlZxThxVEtDgwsamlmH5MRRLfX10LWrCxqaWYfjxFENc+d+XtBwlVXyjsbMrKKcOKrhtttg9mx3U5lZh+TEUQ11dbDBBi5oaGYdkhNHpb36Kowale4Ul/KOxsys4pw4Ks0FDc2sg3PiqKRCQcOBA2GddfKOxsysKpw4Kun++2HaNBc0NLMOzYmjkgoFDfffP+9IzMyqxomjUmbOhDvvhGOPdUFDM+vQqpo4JA2U9JKkiZLOamJ/b0kjJD0raZSkXtn2r0oaW/TzsaSDsn0DJI2R9LykIZI6V/M9lKxQ0NDdVGbWwVUtcUhaBrgM2BfoCxwpqW+jwy4Cro6ILYDzgN8CRMTIiNgqIrYCBgBzgfsldQKGAIMjYjPgDSD/6UsR6d6N/v1d0NDMOrxqtjj6AxMjYlJEzANuAA5sdExf4KHs8cgm9gMcCtwbEXOB7sC8iHg52/cAcEjFIy/X00/D+PG+U9zMakI1E8c6wJSi51OzbcXGAYOyxwcD3SR1b3TMYOD67PHbQGdJ/bLnhwLrNvXikk6R1CCpYebMmUv4FkrkgoZmVkPyHhw/E9hd0jPA7sA0YEFhp6S1gM2B4QAREaREcrGkp4APio8vFhGXR0S/iOjXs2fP6r2DuXPh+uvhsMNg5ZWr9zpmZm1ENQeWp7Foa6BXtu0zETGdrMUhaSXgkIiYVXTI4cDtEfFp0TmPA7tm5+wNfLkawZfs1ltd0NDMako1WxxPAxtJWk/SsqSWwrDiAyT1yAa8Ac4G6htd40g+76YqnLNG9udywE+Bv1ch9tLV1cGGG8Kuu+YahplZa6la4oiI+cBppG6mCcBNETFe0nmSDsgO2wN4SdLLwJrA+YXzJfUhtVgebnTpH0uaADwL3BURD5GXiRPh4Ydd0NDMaorSsEHH1q9fv2hoaKj8hc85B373O5g82bWpzKzDkTQ6Ivo13p734Hj7VShouO++ThpmVlOcOJbU8OEwfbrvFDezmuPEsaTq6qBnT/jGN/KOxMysVTlxLImZM2HYMBc0NLOa5MSxJK65BubPdzeVmdUkJ45yFQoabr89bLpp3tGYmbU6J45yPfUUvPCC7xQ3s5rlxFGu+npYYQU44oi8IzEzy4UTRzk+/NAFDc2s5jlxlOPWW+GDD9xNZWY1zYmjHHV1sNFGsMsueUdiZpYbJ45SvfIKPPKICxqaWc1z4ijVlVdCp05w3HF5R2JmlisnjlLMnw9DhsB++8Haa+cdjZlZrpw4SuGChmZmn3HiKEVdHayxhgsampnhxNGyt96Cu+5KBQ27dMk7GjOz3DlxtMQFDc3MFuHE0ZyIVGJkhx2gb9+8ozEzaxOcOJrz5JMuaGhm1ogTR3Nc0NDM7AucOJqzwQbwgx9At255R2Jm1mZ0zjuANu2nP807AjOzNsctDjMzK4sTh5mZlcWJw8zMyuLEYWZmZalq4pA0UNJLkiZKOquJ/b0ljZD0rKRRknpl278qaWzRz8eSDsr27SlpTLb9UUkbVvM9mJnZoqqWOCQtA1wG7Av0BY6U1Pj264uAqyNiC+A84LcAETEyIraKiK2AAcBc4P7snL8BR2f7rgPOrdZ7MDOzL6pmi6M/MDEiJkXEPOAG4MBGx/QFHsoej2xiP8ChwL0RMTd7HsDK2eNVgOkVjdrMzJpVzcSxDjCl6PnUbFuxccCg7PHBQDdJ3RsdMxi4vuj5ScA9kqYCxwK/q1jEZmbWorxvADwTuFTSN4FHgGnAgsJOSWsBmwPDi875H2C/iHhS0o+BP5GSySIknQKckj2dI+mlJYyxB/D2Ep5bTY6rPI6rPI6rPB01rt5Nbaxm4pgGrFv0vFe27TMRMZ2sxSFpJeCQiJhVdMjhwO0R8Wl2TE9gy4h4Mtt/I3BfUy8eEZcDly/tm5DUEBH9lvY6lea4yuO4yuO4ylNrcVWzq+ppYCNJ60laltTlNKz4AEk9JBViOBuob3SNI1m0m+o9YBVJX86e7wVMqHjkZma2WFVrcUTEfEmnkbqZlgHqI2K8pPOAhogYBuwB/FZSkLqqTi2cL6kPqcXycKNrngzcKmkhKZF4hSUzs1ZU1TGOiLgHuKfRtv8tenwLcMtizn2dLw6mExG3A7dXNNDmLXV3V5U4rvI4rvI4rvLUVFyKiGpc18zMOiiXHDEzs7I4cZiZWVmcOBZD0rqSRkp6QdJ4SafnHROApOUlPSVpXBbXr/KOqUDSMpKekXR33rEUk/S6pOey+mYNecdTIGlVSbdIelHSBEk7toGYNm5UJ262pDPyjgtA0v9k/+afl3S9pOXzjglA0ulZTOPz/LuSVC/pLUnPF21bXdIDkl7J/lytEq/lxLF484EfRURfYAfg1CZqbeXhE2BARGwJbAUMlLRDviF95nTa7vTor2b1z9rSXPu/APdFxFeALWkDf3cR8VJRnbhtSXXiWnMySpMkrQP8AOgXEZuRZmoOzjcqkLQZcDKpxNKWwDdyLLx6FTCw0bazgBERsREwInu+1Jw4FiMiZkTEmOzxB6T/1F+Y5dXaIpmTPe2S/eQ+wyGrbPx14Iq8Y2kPJK0C7AbUAUTEvEY3v7YFewKvRsQbeQeS6Qx0ldQZWIG2UaduE+DJiJgbEfNJtw8MauGcqoiIR4B3G20+EBiSPR4CHFSJ13LiKEF2T8nWwJMtHNoqsi6hscBbwANFd9Ln6c/AT4CFOcfRlADulzQ6K0XTFqwHzASuzLr3rpC0Yt5BNdK4TlxuImIaqZr2ZGAG8H5E3N/8Wa3ieWBXSd0lrQDsx6IVM/K2ZkTMyB7/F1izEhd14mhBVgrlVuCMiJiddzwAEbEg60roBfTPmsu5kfQN4K2IGJ1nHM3YJSK2IZX4P1XSbnkHRPr2vA3wt4jYGviQCnUjVEJW7eEA4Oa8YwHI+uYPJCXctYEVJR2Tb1QQEROA35OWfbgPGEtRvb22JNK9FxXpnXDiaIakLqSkMTQibss7nsayro2RfLFfs7XtDBwg6XVS+fwBkq7NN6TPZd9WiYi3SP31/fONCEjVoqcWtRZvISWStmJfYExEvJl3IJmvAa9FxMysdt1twE45xwRARNRFxLYRsRupmsXLecdU5M2sWGyhaOxblbioE8diSBKp/3lCRPwp73gKJPWUtGr2uCupXteLecYUEWdHRK+I6EPq3ngoInL/NgggaUVJ3QqPgb1J3Qu5ioj/AlMkbZxt2hN4IceQGmtcJy5vk4EdJK2Q/d/ckzYwmQBA0hrZn18ijW9cl29EixgGHJ89Ph64sxIXzbuselu2M2m9j+ey8QSAn2VlVPK0FjAkW2GxE3BTRLSp6a9tzJrA7emzhs7AdRHRZEXlHHwfGJp1C00CvpVzPMBnCXYv4Nt5x1KQLaNwCzCGNOPxGdpOmY9bs3WEPgVOzWuSg6TrSfX/eiitV/QL0npFN0k6EXiDVHF86V/LJUfMzKwc7qoyM7OyOHGYmVlZnDjMzKwsThxmZlYWJw4zMyuLE4dZBUnaI8/qwJK+KenSvF7faoMTh5l9Jrs/yKxZThxWcyQdk61pMlbSPwoflpLmSLo4W1dhhKSe2fatJD0h6VlJtxfWNJC0oaQHs7VRxkjaIHuJlYrW2Ria3encOIZRkn6fxfGypF2z7Yu0GCTdLWmPovj+kMX3oKT+2XUmSTqg6PLrZttfkfSLEt/3HyWNA3JfE8TaPicOqymSNgGOAHbOCkUuAI7Odq8INETEpqTy2IUP3auBn0bEFsBzRduHApdla6PsRKraCqmS8hlAX2B9UhWCpnSOiP7Zsb9YzDHFViSVc9kU+AD4DekO74OB84qO6w8cAmwBHCapXwnv+8mI2DIiHi0hDqtxLjlitWZP0gJFT2cNga58XvhtIXBj9vha4LZs3YxVI+LhbPsQ4Oas/tU6EXE7QER8DJBd86mImJo9Hwv0AZr6QC4UzhydHdOSeaQKrJAS2CcR8amk5xqd/0BEvJO9/m3ALqQyHYt73wtIxTzNSuLEYbVGwJCIOLuEY5e0Hs8nRY8XsPj/Z580ccx8Fu0JKF4e9dP4vEbQwsL5EbEwW9yooHHcQfPv++OIaJOlwK1tcleV1ZoRwKFFFU1Xl9Q729cJODR7fBTwaES8D7xXGIMgFb58OFsVcqqkg7LrLJct5LO0Xge2ktRJ0rosWQn4vbL31ZW04ttjNP++zcriFofVlIh4QdK5pBUBO5FVNCVVDv2QtDDWuaRunCOy044H/p4lhuIqtscC/5B0XnadwyoQ4mPAa6QS6xNI1WDL9RSp66kXcG1ENAA0877NyuLquGYZSXMiYqW84zBr69xVZWZmZXGLw8zMyuIWh5mZlcWJw8zMyuLEYWZmZXHiMDOzsjhxmJlZWf4/xDlINkcRrx8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "def eval_show(epoch_per_eval):\n",
    "    plt.xlabel(\"epoch number\")\n",
    "    plt.ylabel(\"Model accuracy\")\n",
    "    plt.title(\"Model accuracy variation chart\")\n",
    "    plt.plot(epoch_per_eval[\"epoch\"], epoch_per_eval[\"acc\"], \"red\")\n",
    "    plt.show()\n",
    "    \n",
    "eval_show(epoch_per_eval)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "从上图可以一目了然地挑选出需要的最优模型权重参数`ckpt`文件。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 总结"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "本例使用MNIST数据集通过卷积神经网络LeNet5进行训练，着重介绍了利用回调函数在进行模型训练的同时进行模型的验证，保存对应`epoch`的模型权重参数`ckpt`文件，并从中挑选出最优模型的方法。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "MindSpore-1.0.1",
   "language": "python",
   "name": "mindspore-1.0.1"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.5"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
